Choosing the lesser evil: trade-off between false discovery rate and non-discovery rate

The problem of multiple comparisons has become increasingly important in light of the significant increase in volume of data available to statisticians. The seminal work of Benjamini and Hochberg (1995) on the control of the false discovery rate (FDR) has brought forth an alternative way of measuring type I error rate that is often more relevant than the one based on the family-wise error rate. In this paper, we emphasize the importance of considering type II error rates in the context of multiple hypothesis testing. We propose a suitable quantity, the expected proportion of false negatives among the true alternative hypotheses, which we call non-discovery rate (NDR). We argue that NDR is a natural extension of the type II error rate of single hypothesis to multiple comparisons. The utility of NDR is emphasized through the trade-off between FDR and NDR, which is demonstrated using a few real and simulated examples. We also show analytically the equivalence between the FDR-adjusted p-value approach of Yekutieli and Benjamini (1999) and the q-value method of Storey (2002). This equivalence dissolves the dilemma encountered by many practitioners of choosing the “right” FDR controlling procedure.